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ABSTRACT 

The Propulsion Checkout and Control System (PCCS) is a 
predictive maintenance software system. The real-time 
checkout procedures and diagnostics are designed to detect 
components that need maintenance based on their condition, 
rather than using more conventional approaches such as 
scheduled or reliability centered maintenance. Predictive 
maintenance can reduce turn-around time and cost and 
increase safety as compared to conventional maintenance 
approaches. Real-time sensor validation, limit checking, 
statistical anomaly detection, and failure prediction based 
on simulation models are employed. Multi-signal models, 
useful for testability analysis during system design, are used 
during the operational phase to detect and isolate degraded 
or failed components. The TEAMS-RT real-time diagnostic 
engine was developed to utilize the multi-signal models by 
Qualtech Systems, Inc. 

Capability of predicting the maintenance condition was 
successfully demonstrated with a variety of data, from 
simulation to actual operation on the Integrated Propulsion 
Technology Demonstrator (IPTD) at Marshall Space Flight 
Center (MSFC). Playback of IPTD valve actuations for 
feature recognition updates identified an otherwise 
undetectable Main Propulsion System 12 inch prevalve 
degradation. The algorithms were loaded into the 
Propulsion Checkout and Control System for further 
development and are the first known application of 
predictive Integrated Vehicle Health Management to an 
operational cryogenic testbed. The software performed 
successfully in real-time, meeting the required performance 
«oal of 1 second cycle time. 

1. PROJECT DESCRIPTION 
1,1 IPTD Overview 

The Systems Health Management Group from Ames Research 
Center (ARC), Code IC, participated on a product 
development team led by Rockwell Space Systems Division 
(now Boeing North American) during Phase 1 of the X-33 


program for the Integrated Propulsion Technology 
Demonstrator located at MSFC. NASA team members 
included MSFC (test article integration and test 
operations), ARC (onboard diagnostic/prognostic 
algorithms and fault isolation models) and Lewis * Research 
Center (LeRC) (smart sensor algorithms). The IPTD 
objective was to evaluate and predict ground and flight 
propulsion operations improvements w r hich reduce turn- 
around-time and operations costs risk. 

5.2 PCCS Overview 

The PCCS controls the X-33 Main Propulsion System 
(MPS) type components on the IPTD and monitors 
hundreds of measurements from the associated sensors. The 
PCCS consists of nine modules: 1) the Task Wrapper, 2) the 
Task Manager, 3) Sensor Validation, 4) System Transient 
Detector, 5) Sensor Reconstruction, 6) Feature and Level 
Detection with Prognostics and Limit Checking, 7) Valve 
Function and Sensor Validation, 8) Failure Isolation 
(TEAMS-RT 1 ), and 9) Significant Event Record (SER) 
Generator. The software runs on a SPARC 20 workstation. 
ARC developed modules 2, 6, 7, and 9 and integrated them 
with modules 3, 4, and 5 from LeRC and module 8 from 
Qualtech Systems, Inc. The Task Manager interfaces w ith 
the Task Wrapper, developed at Rockwell, which provides 
the interfaces to the test stand. A top level view of these 
modules is shown in Figure L 

There were 3 major goals in this task: 

1) Verification of Reusable Launch Vehicle (RLV) on- 
board propulsion prognostic and diagnostic 
algorithms on the IPTD. 

2) Demonstration of the use of design models in real-time 
system monitoring and fault isolation. 

3) Demonstration of real-time performance of PCCS on 
IPTD to reduce the risk of deploying such a system for 
actual operations. 


1 The Testability Engineering and Maintenance System - 
Real-time (TEAMS-RT) is available from Qualtech Systems 
Inc., 66 Davis Road, Storrs, CT 06268 at www .teamqsi.com. 




Figure 1: PCCS Top Level Architecture Over iew 


The following sections describe the software architecture of 
the PCCS and the results of the IPTD test stand 
demonstrations that occurred at the end of this task. 

2. PCCS ARCHITECTURE 

2. 1 Architecture Overview 

A brief description of the eight software modules developed 
or integrated by ARC follows: 

1) Task Manager - controls the sequencing of the analysis 
modules. 

2) Sensor Validation - detects sensor failures. 

3) System Transient Detector - detects transient features 
which occur during periods of steady operation and 
which are not attributed to sensor failure or normal 
system operation. 

4) Sensor Reconstruction - calculates a replacement value 
for a failed sensor based on models and on the values of 
nominal sensors. 

5) Feature and Level Detection with Prognostics and 
Limit Checking - detects features and measured levels 
in the signals acquired from sensors on the mechanical 
components, predicts the future values of the features 
and levels, checks the current and future values of the 
features and levels against pre-defined limits, and flags 
maintenance and failure warnings. 

6) Valve Function and Sensor Validation - determines 
which system sensors have changed their status to 
failed or reconstructed. 

7) TEAMS-RT - uses model-based reasoning to determine 
the location of components which have failed or need 
maintenance using feature and level information. 

8) Significant Event Record Generator - produces the 
SERs to be used for determining maintenance actions. 


2.2 ARC Module Overviews 

2.2.1 Task Manager (TM) 

The Task Manager interfaces to the Task Wrapper, and hence, 
the teststand. and passes along sensor data and status from 
the Task Wrapper to the other modules. The Task Manager 
controls the sequencing of the analysis modules and 
outputs sensor and valve status and Significant Event 
Records to the Task Wrapper which then displays 
appropriate status messages to the operator. 

2.2.2 Feature and Level Detection with Prognostics and 
Limit Checking (FLDPLC) 

The Feature and Level Detection module detects features and 
measures levels in the signals acquired from sensors on the 
mechanical components, predicts the future value of the 
features and levels, checks the current and future values 
against predefined limits, and flags maintenance and failure 
warnings. Depending on the mode of the PCCS, certain 
sensor values are used to infer more information about some 
particular component of the system. A group of sensor 
values is used to determine a feature. A feature may*be peaks, 
rise time, fall time, etc. of a wave form. This feature is then 
compared to predefined limits. If it is out of bounds a failure 
warning is issued. The result is also used to predict when a 
limit may be reached in the future. A maintenance w-aming is 
issued if the limit is likely to be reached within a prescribed 
time interval. 

2.2.3 Valve Function and Sensor Validation (VFSV) 

The Valve Function and Sensor Validation module 
determines which system sensors have changed their status 
to failed or reconstructed and outputs a list of the changed 
sensors. System sensor status is determined outside of this 
module. VFSV sends results of its sensor status continuity 
analysis to the SER Generator. 

2.2.4 Significant Event Record Generator (SERG) 

The Significant Event Record Generator creates SERs from 
the results of the analysis modules, FLDPLC, TEAMS-RT, 
SV, and VFSV. These SERs are used by Informed 
Maintenance (IM) modules for determining required 
maintenance actions. 

2.3 ARC Input File Descriptions 

Many of the input files utilized by these modules are 
developed at Rockwell and delivered to ARC as a Microsoft 
Excel workbook. The individual sheets are then reformated 
as text files. The information in these files is used to 
establish the appropriate system configuration according to 
the current phase, function, and command from the test stand. 

2.4 Qualtech Module: TEAMS-RT 

The TEAMS-RT module implements a model based 
reasoning approach, wherein information about failure 
sources, tests and monitoring points, redundancy and 




system modes are captured in colored directed graph models 
known as multi-signal models [1,2]. In simple terms, these 
models enable the inference engine to interpret test results 
by answering these questions: given a test Tl, which 
components can cause it to fail; or, if the health of 
component Cl needs to be checked, which tests observe Cl. 
Such models may be automatically generated via fault 
simulation (using simulators such as Saber. PSpice [3], 
MATRIX* or VHDL simulators) or entered manually in 
TEAMS, the model development tool, based on an 
engineering understanding of the system or legacy data 
captured in FMECA reports, fault trees. CAD data, and 
technical documentation. The same model is then used by 
TEAMS-RTfor real-time monitoring, thus ensuring that the 
results predicted in the design stage by TEAMS are indeed 
achieved in actual application. In our case, the multi-signal 
models were created in TEAMS at Rockwell. 

The objective of the TEAMS-RT inference engine is to 
associate four distinct (failure) states with each component 
in the system: (1) Good . (2) Bad. (3) Suspected, and (4) 
Unknown . In addition, results of prognostics checking (see 
Section 2.2.2) were used to diagnose components that w-ere 
Suspected to be in need of maintenance and Definitely in 
need of maintenance . TEAMS-RT also has additional 
capabilities of handling dynamic system mode changes, and 
diagnosis and prognosis in fault-tolerant systems with 
built-in redundancy. Some unique features of TEAMS-RT 
are: 

(i) efficient real-time processing of sensor results; 

(ii) update of fault-test point dependencies in response to 
system mode changes; 

(iii) update of dependencies resulting from failures in 
redundant components; 

(iv) and detection and isolation of multiple simultaneous 
failures. 

The TEAMS-RT algorithms developed for the IPTD 
demonstration were extensively tested in a simulation 
environment. Multiple faults were seeded in randomly 
generated models. These models were then exercised in 
different modes of operation and pass/fail results of tests 
were fed to TEAMS-RT. Table 1 presents simulation results 
for TEAMS-RT on a 1000x1000 system with 80 modes of 
operation. Column 1 lists the number of faults inserted. |Tp| 
is the number of tests that passed in spite of the failures. The 
remaining columns list the number of components that were 
declared to be good, bad and suspected (residual ambiguity) 
by TEAMS-RT, and the processing time. Thus, the 
simulation results indicated that TEAMS-RT will easily 
satisfy the design goal of processing 1000 test results in 
under 200 msec. 

3. MSFC TESTS AND RESULTS 

3.1 Test Description 

The Informed Maintenance demonstration tests were 
conducted on the Integrated Propulsion Technology 


Demonstrator at MSFC. The propulsion module of the IPTD 
consists of propulsion system hardware and the Propulsion 
Checkout and Control System. The propulsion hardware 
components are integrated within a structural support 
fixture which includes mounting provisions for the system 
feed lines, liquid oxygen (LO2) and hydrogen (LH2) tanks, 
vents, and multiple valves. The system is installed in the 
MSFC Advanced Engine Test Facility (AETF). The PCCS 
provides the hardware and supporting software to checkout 
and control the IPTD. The hardware includes a smart 
controller, multiple sensors, and data acquisition system. 
The software provides signal processing, data analysis, 
automated reasoning, and decision making. Sensor data 
flows to the software once per second, which establishes the 
performance requirement for the software system of 1 second 
cycles. 


Table 1: Performance results of TEAMS-RT for 
simulated system with 1000 faults and tests. 


faults 

l T Pl 

Good 

Bad 

Suspect 

time(ms) 

1 

993 

997 

1 

2 

50 

2 

978 

996 

j 2 

2 

i£Q 

5 

931 

991 

5 

4 

50 

10 

881 

983 

10 

7 

75 

20 

819 

973 

20 

7 

87 


The mechanical components of thv IPTD are monitored via 
the sensors. These may include observations of temperature, 
pressure, electrical current, or other physical phenomena. 
Information from the sensors is digitized into time series 
data. The data are processed via selected signal processing 
and conditioning algorithms to extract features of interest. 
The automated reasoning and decision software, TEAMS- 
RT, is used for system fault diagnosis. 

The IM demonstration tests were performed during a two 
month period, from late March to May 1996. At the 
beginning, the hardware was integrated and was checked 
out by data acquisition software. The digitized sensor data 
for each test case were saved in data files, then the data files 
were processed and analyzed off-line. Using data from many 
test cases, it was determined that the hardware was 
integrated properly and all parts of the software were 
working correctly. There were three types of software tests. 
1) tests with data of natural degradation, 2) tests with data 
of injected degradation, and 3) tests with simulation data. In 
the final week of testing, the full PCCS software system w as 
tested with live data. These tests are described in detail in 
the following section. 

3.2 Test Results 

3.2.1 Tests with Data from Natural Component 
Degradation 

During the test stand operations, PV202, a prevalve, 
showed a natural degradation in performance and slammed 
on a valve closing operation. Pressure signature patterns for 




PV202 Close Pressure Slgnslures 


the valve as the valve degraded are shown in Figures 2 and 
3. When data prior to the slamming were input to the 
software, it computed several features. The FLDPLC 
correctly predicted that the initial actuation pressure value 
would be outside the nominal range within the next 4 valve 
closing operations (Figure 4), and recommended 
maintenance for the valve. 

3.2.2 Tests with Data of Injected Degradation 

In the last week of the test period (May 20 - May 24), the 
software and hardware were integrated and several 
automated tests were performed with everything operational 
in real-time at MSFC AETF. 

According to the IM test plan, eight test cases were 
performed. The results of two test cases are reported here. 

3.2.2. 1 Basic Test 

Test Description: 

Cycle PV201 (PreValve), PI '2 03 (SpringValve), and 
PV204 ( SpringValve ) five times in 02_PREJTEST phase 
without any changes or adjustments. 

The test results showed that the software had been 
integrated properly. The computer running time for data 
acquisition, signal processing, and feature extraction was 
less than 10% of the requirement. The baseline measurements 
of features were obtained, and the fact that there was 
negligible difference between different measurements for the 
same features showed that the PCCS w'as stable and accurate. 

3. 2.2. 2 Fault Injection Test on PV203 in 02_PRE_TEST 
Phase. 

Test Description: 

Cycle PV203 while adjusting the needle valve from a 
healthy PV203 operation until a failure event is generated . 
This is to see if the orifice gets flagged as the bad 
component . 

The test results showed that the actuation time and the total 
transition time for the Spring Valve were getting longer as 
the needle valve was adjusted to simulate more clogging, as 
expected. The software predicted a failed component, the 
orifice, and asked for maintenance for the valve. The values of 
the corresponding transition times and actuation transition 
times computed in the FLDPLC are summarized in Table 2. 



Figure 2: PV202 Close Pressure Signatures 



Figure 3: PV202 Open Pressure Signatures 
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Figure 4: PV202 Maintenance Recommendation 





Table 2: Summary of Feature Values for PV 203 Tests 


PV 203 Test 

Transition 
Time (sec) 

Actuation 
Transition Time 
(sec) 

p\203 31 good 

3.06 

0.9 ] 

dy 203 31 5t 

7.52 

2.26 

dy 203 31 3t 

10.14 

3.12 

d\ 203 31 2t 

16.44 

5.24 


3.2.3 Tests with Simulation Data 

MATRIX x : simulation models were available for the pre- 
valve. spring valve and solenoid valve. Simulation runs 
were used to understand the behavior of components. In the 
case of the prevalve, simulation runs were used to decide 
which features would be diagnostic of which degradations. 
Prevalve simulation data were also used to test the software 
before testing on the MSFC test stand. 

3.2.4 Tests of the TEAMS-RT Diagnostic Capability 

The multi-signal model of the IPTD was developed in two 
stages. First, the basic topology of the teststand 
components was modeled. Valves that controlled propellant 
flow w r ere isolated as the major functions. Components that 
control or support these valves were grouped with the 
valve. Signals were associated with each valve and then 
tests were defined for the signals. Flow characteristics were 
then modeled for a cryogenic valve. This model included 
temperature, pressure, and flow rate, in both the forward and 
backward directions for each operating mode of the valve. 
Tests were added for the system parameters, temperature and 
pressure transients, based on what measurements were 
available on the test stand. 

Once the multi-signal model was constructed, the model was 
placed in different operational modes and tests were selected 
for different operational modes. The propagation of signals 
was checked to see if the dependency relationships were 
properly defined. The real-time system test did not proceed 
as. easily, however. Fault isolation software could not be 
fully tested because there were basic errors made in labeling 
the nodes in the model, which lead to misdiagnosis. It was 
not possible to rectify the model for the tests with live data 
from the IPTD within the allocated week of test time. Our 
recommendation is for earlier completion of the TEAMS 
models so that checkout could occur with simulation or off- 
line test stand data, enabling obvious modeling errors to be 
corrected. It is expected that the basic models would also 
need later revisions as behavior of the interactions of 
components is better understood from observing system 
operation. 


2 MATRIX x is available from Integrated Systems, Inc., 201 
Moffett Park Dr., Sunnyvale, CA 95054-3309, phone (408) 
542-1575. 


4. SUMMARY AND CONCLUSIONS 

In summary, PCCS software modules were tested with a 
variety of data, from simulation to actual operation on the 
Integrated Propulsion Technology Demonstrator at MSFC. 
Capability of predicting the maintenance condition was 
successfully demonstrated with all data. Playback of IPTD 
valve actuations for feature recognition updates identified 
an otherwise undetectable Main Propulsion System 12 inch 
prevalve degradation. The algorithms have been loaded into 
the IPTD for further development and are the first known 
application of predictive Integrated Vehicle Health 
Management to an operational cryogenic testbed. The 
software performed successfully in real-time, meeting the 
required performance goal of 1 second cycle time. 
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